Discovering some regression bugs that degrade the performance or behavior of software is difficult without a production workload. A production environment may be available when a service progresses through the release cycle all the way to a canary release or until the service is fully deployed in production. A revision of source code that is committed to version control may cause a regression in performance that is only evident under high load or reveal that a latent bug that remained hidden for a long time is now triggered. Generally, these kinds of defects are difficult to isolate. For example, it can take longer to isolate a bug in source code than it takes to fix the bug. When a latent bug later emerges, some context regarding the offending commit may be lost, which can increase the time needed to fix the bug. A performance regression is important to quickly discover and isolate because it can decrease end user satisfaction and increase the consumption of valuable resources, such as time and electricity.
Here are some reasons why finding regressions late in release cycle is problematic:                Isolating offending commit(s) is hard;        Even after the commit has been identified. The semantics of the code or the logical flow might be forgotten, and it might be difficult to fix the bug without completely understanding the intent of the entire commit;        The code needs to roll back from Production machines;        The release cycle is reset and throws off team timelines and deliverable schedules;        New developers get skittish about making check-ins, perhaps because a release may be difficult to roll back. This can cause stress and impact the job satisfaction of a developer.        Release cycles are deliberate and long, exacerbating the issues mentioned above. A way is needed to handle these issues before the service gets released.        